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Abstract.  This  paper  presents  a  model-assisted  probability  of  detection  (MAPOD)  study  for  inspection  of 
a  two-layer  airframe  structure.  Eddy  current  measurements  for  varying  crack  length  around  fastener  holes 
in  a  two-layer  aluminum  structure  are  studied  using  both  experimental  and  model-generated  data.  New 
statistical  algorithms  are  used  to  calculate  the  probability  of  detection.  Good  agreement  was  achieved 
between  empirical  and  model-assisted  approaches. 

Keywords:  model-assisted  probability  of  detection  (MAPOD),  eddy  current 
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INTRODUCTION 

The  United  States  Air  Foree  (USAF)  antieipates  dramatie  inereases  of  inspeetions 
required  for  maintaining  the  aging  aireraft  fleet;  therefore,  faster  and  cheaper 
implementation  of  new  inspection  technologies  is  required.  The  reliability  of  these  new 
inspection  technologies  will  be  assessed  by  a  probability  of  detection  (POD)  study.  POD  is 
a  metric  used  to  quantify  the  reliability  of  inspection  systems.  POD  provides  engineers  with 
the  minimum  flaw  size  that  can  be  reliably  detected  and  can  also  be  used  as  an  input  for 
probabilistic  risk  assessment  analysis.  Traditional  POD  evaluation  methodologies  are 
entirely  empirical.  In  most  cases,  the  cost  of  manufacturing  the  number  of  samples  required 
for  a  traditional  POD  study  is  prohibitive  and  may  delay  or  prevent  a  new  inspection 
procedure  or  new  technology  from  being  implemented.  Alternative  methodologies  have 
been  proposed  that  incorporate  modeling  to  reduce  the  number  of  samples  required. 
Statistical  analysis  methods  have  also  evolved  since  the  standard  method  for  conducting 
POD  studies  was  codified  in  MIL-HDBK-1823.  This  study  investigates  a  class  of  aerospace 
structural  inspection  problems  and  considers  the  use  of  models  and  new  statistical  analysis 
methods  in  POD  evaluation. 

PRIOR  WORK 

The  concept  of  using  models  to  assist  in  quantifying  the  performance  of  eddy  current 
inspection  systems  has  been  proposed  and  used  with  some  success  in  the  past.  Martinez  et 
al.  used  basic  1-D  analytical  models  and  limited  experimental  signal-to-noise  data  to 
compare  different  eddy  current  systems  [1].  This  work  was  limited  to  air-cored  probes  and 
flat  plate  geometries  with  surface  breaking  notches.  Beissner  et  al.  and  Nakagawa  used 
more  sophisticated  models  based  on  the  boundary  element  method  to  predict  the  POD  for 
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different  3-D  surface  breaking  notches  [2,3]-  Again,  limited  experimental  data  was  used  to 
characterize  and  incorporate  the  signal-to-noise  ratio.  For  more  examples  of  model-assisted 
POD  work,  the  reader  is  directed  to  the  papers  by  Gray  et  al.  and  Thompson  [4,  5] 

EXPERIMENTAL  STUDY 

For  this  study,  two-layer  aluminum  specimens  were  used.  Each  specimen  contains  10 
countersunk  fasteners.  90%  of  the  fasteners  are  titanium  and  10%  are  steel.  The  steel 
fastener  sites  were  excluded  from  this  study.  All  fatigue  cracks  are  at  the  top  or  bottom  of 
the  fastener  site  (i.e.  they  are  not  in  between  the  fastener  sites).  A  subset  of  specimens  with 
cracks  located  in  the  second  layer  at  the  faying  surface  was  selected  for  use.  Crack  lengths 
range  from  0.027”  (0.69  mm)  to  0.169”  (4.29  mm).  There  were  a  total  of  171  fastener  sites 
in  the  study  and  38  of  the  fastener  sites  had  cracks.  A  reflection  eddy  current  probe 
operating  at  600  Hz  was  used  throughout  the  study.  Teflon  was  placed  in  between  the  probe 
and  the  sample.  Both  the  in-phase  and  quadrature  components  of  the  voltage  were  recorded 
in  the  format  of  a  C-scan  with  a  scan  step  size  of  0.010"  (0.254  mm).  Interpolation  was 
used  to  determine  the  voltage  at  points  along  different  radii  around  the  fastener  site.  The 
quadrature  component  of  the  voltage  at  a  fixed  distance  from  the  center  of  the  hole  was 
extracted  and  plotted  as  a  function  of  the  swept  angle  around  the  fastener  site  [6].  In  cases 
where  a  large  enough  crack  is  present,  a  perturbation  in  the  form  of  a  localized  Gaussian 
response  appears  as  shown  in  Figure  1(b).  This  peak  measure  was  used  for  the  signal 
response  (a)  throughout  the  study. 

MODEL- ASSISTED  POD  (MAPOD)  APPROACH 

Two  protocols  have  been  proposed  to  incorporate  model-assisted  approaches:  1)  transfer 
function  (TF)  and  2)  full-model  assisted  (FMA)  [7].  This  study  investigates  a  modified 
FMA  approach.  A  protocol  proposed  by  Bruce  Thompson  with  inputs  from  the  MAPOD 
working  group  was  followed  in  this  work.  The  protocol  consists  of  1 1  steps  designated  as 
follows  [8]: 

1)  Identify  the  scope  of  the  POD  study, 

2)  Identify  factors  that  control  signal  and  noise, 

3)  Evaluate  quality  of  physics-based  models, 

4)  Acquire,  develop,  and  validate  simulation  tools, 

5)  Acquire  input  parameters  and  parameter  distributions. 


angular  location  (degrees)  angular  location  (degrees) 

FIGURE  1.  (a)  Response  from  unflawed  titanium  fastener  site  in  a  two-layer  aluminum  structure  and 
(b)  perturbed  response  due  to  the  presense  of  a  fatigue  crack. 
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6)  Conduct  flaw  signal  distribution  simulations  and  noise  signal  distribution  simulations, 

7)  Acquire  remaining  information  on  faetors  empirieally, 

8)  Acquire  marginal  information  on  independent  faetors, 

9)  Acquire  covarianee  information  on  dependent  factors, 

10)  Combine  6,  8,  and  9  into  full  signal  and  noise  distributions, 

1 1)  Compute  POD,  POFC,  and  ROC. 

This  protoeol  is  still  in  development  and  is  subjeet  to  ehange  by  the  time  of  publication  of 
this  paper.  Some  deviations  from  a  strictly  full-model  assisted  approaeh  were  made.  For 
example,  empirieal  data  were  used  to  quantify  noise  and  engineering  judgment  was  applied 
to  estimate  the  effect  of  crack  closure  and  crack  aspect  ratio.  Human  factors,  which 
typically  have  a  tremendous  impact  on  POD  have  been  eliminated  from  this  study  by  using 
an  automated  data  eollection  system  and  relying  on  software  to  extraet  and  classify  features 
from  the  data  pertinent  to  this  partieular  inspeetion. 

MAPOD  DEMONSTRATION 

The  first  step  in  the  FMA  protocol  is  to  identify  the  seope  of  the  POD  study.  The  goal 
of  this  work  is  to  establish  the  feasibility  of  a  MAPOD  evaluation  for  struetural  applieations. 
Eddy  current  will  be  used  exclusively  in  this  study.  The  problem  of  interest  is  the  detection 
of  cracks  under  installed  eountersunk  fasteners  in  the  seeond  layer  of  airframe  structures. 
Comparison  of  empirical  POD  results  with  model-assisted  results  will  determine  the 
feasibility  of  MAPOD  for  this  class  of  problems. 

Factors  that  control  signal  and  noise  inelude  the  NDE  technique,  part  material,  part 
geometry,  and  flaw  eharaeteristics.  For  this  problem,  the  key  factors  to  include  in  the 
evaluation  were  identified  from  experience  with  the  inspection  problem  and  experimental 
studies.  The  probe  eharacteristic  response,  lift-off,  and  sean  resolution  are  important  faetors 
concerning  the  NDE  measurement.  The  surfaee  condition  of  the  samples,  thickness  of  the 
layers,  type  of  fastener,  fastener  fit,  and  distance  between  adjacent  fasteners  and  edges  are 
identified  as  signifieant  factors  related  to  part  geometry  and  material.  The  dimensions, 
aspeet  ratio,  location  around  fastener  site,  and  morphology  are  important  flaw  eharaeteristics 
to  be  considered.  Evaluation  of  the  quality  of  the  physics  based  models  has  been  the  focus 
of  previous  work  [9].  VIC-3D®  has  the  eapability  to  handle  the  generic  two-layer  structural 
problem  of  interest,  is  eomputationally  effieient,  and  was  used  for  this  study  [10].  Other 
faetors  not  eonsidered  in  this  study  include  eracks  at  other  loeations  around  the  fastener  site 
and  multiple  craeks  around  a  single  fastener  site. 

Quantitative  data  on  the  range  of  expected  values  for  all  of  the  factors  that  control  signal 
to  noise  are  neeessary  for  the  input  parameters  to  the  simulations  and  experimental  studies. 
Some  of  these  parameters  are  known,  others  need  to  be  determined  empirically.  In 
particular,  conductivity  of  the  materials,  eoil  dimensions,  liftoff,  and  scan  resolution  were 
known  before  the  experiment  was  condueted.  The  only  parameter  known  about  the  flaws 
was  the  length.  The  probe  eharacteristic  response,  measurement  noise,  and  surface 
condition  (paint)  needed  to  be  determined  empirieally.  Some  input  parameters  sueh  as  eraek 
aspeet  ratio  and  eraek  morphology  were  unknown  and  eould  not  be  determined  empirically, 
so  approximations  were  made  based  on  engineering  judgment.  In  future  work,  information 
from  destruetive  tests  may  be  useful  in  making  deeisions  about  unknown  input  parameters. 

The  next  steps  (6-10)  were  addressed  through  a  series  of  experimental  and  simulated 
studies  evaluating  the  sensitivity  of  the  key  factors  on  signal  and  noise.  The  probe  was  first 
characterized  using  a  ealibration  proeedure  developed  in  previous  work  to  effieiently  relate 
simulated  impedance  ealeulations  with  measured  voltage  data  [11].  A  nonlinear  least 
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squares  estimator  (NLSE)  is  used  to  determine  two  eomplex  parameters  related  to  gain  and 
bias  which  are  then  used  to  provide  a  transformation  from  the  model  generated  impedance 
data  to  the  experimental  voltage  data  or  vice  versa.  Five  unflawed  hole  responses  for 
titanium  were  taken  from  the  experimental  data  and  then  averaged.  These  two  complex 
parameters  remained  constant  throughout  the  rest  of  the  study.  The  results  are  displayed  in 
the  Figure  2.  Since  the  basic  shape  of  the  simulated  data  matches  the  shape  of  the  empirical 
data,  there  is  good  reason  to  be  confident  that  this  transformation  was  appropriate. 

Simulations  were  then  conducted  according  to  the  method  reported  in  [12].  VIC-3D® 
was  used  for  all  simulations  [13].  All  measurable  parameters  for  the  probe  and  samples 
were  included  in  the  model.  The  dimensions  0.156”  (3.96  mm)  and  0.100”  (2.54  mm)  for 
the  top  and  bottom  layer  respectively.  A  conductivity  of  (1.87  xlO^  S/m)  for  the  aluminum 
was  used  in  the  model.  A  conductivity  of  (1.79  xlO^  S/m)  was  used  for  the  titanium 
fasteners.  The  diameter  of  the  fastener  was  0.025”  (0.635  mm)  and  had  a  100  degree  cone 
head.  The  distance  between  each  hole  is  0.73”  (18.5  mm).  A  diameter  of  0.1 18”  (3.0  mm) 
and  0.236”  (6.0  mm)  for  the  inner  and  outer  radius  respectively  was  used  for  probe 
parameters.  A  coil  height  of  0.236”  (6  mm)  was  used  and  a  ferrite  cup  core  with  a 
permeability  of  1000  was  included.  Two  flaw  conditions  were  simulated:  a  through  notch 
and  a  comer  notch.  The  aspect  ratio  of  the  comer  notch  was  assumed  to  be  1 : 1 ,  so  it  follows 
that  the  notch  will  be  a  through  notch  when  the  length  of  the  notch  exceeds  0.100”.  Since 
little  information  is  available  on  the  range  of  crack  aspect  ratios  to  expect  in  multi-layer 
stmctures,  the  choice  of  1 : 1  was  made  for  simplicity.  Destmctive  studies  to  determine  the 
distribution  of  the  aspect  ratio  of  cracks  in  multi-layer  stmctures  could  improve  the  model- 
assisted  POD  process.  For  this  study,  all  cracks  were  treated  as  notches,  so  it  assumed  that 
there  is  no  electrical  contact  between  crack  faces.  Again,  this  is  a  simplification  and  more 
studies  would  be  beneficial.  After  the  simulations  were  conducted,  the  feature  extraction 
technique  described  in  the  experimental  section  was  used  to  determine  the  value  of  a  [6]. 

The  results  of  the  simulations  along  with  the  experimental  data  are  displayed  in  Figure 
3.  Although  there  is  good  qualitative  agreement  between  the  simulated  data  and  the 
empirical  data,  there  are  weaker  than  expected  responses  for  some  of  the  larger  cracks.  This 
is  likely  due  to  variation  in  crack  aspect  ratio  or  crack  morphology.  The  unflawed  data  is 
clearly  visible  at  flaw  sizes  equal  to  zero  in  Figure  4(a)  and  the  probability  density  of  the 
unflawed  responses  is  shown  in  Figure  4(b).  The  noise  has  a  basic  Gaussian  distribution. 
To  analyze  this  data,  a  threshold  was  set  at  -0.02.  Only  data  above  this  value  was  used  in 
the  analysis.  A  single  outlier  for  the  unflawed  case  was  also  removed  from  the  analysis.  It 
is  apparent  that  the  data  above  the  threshold  does  not  have  a  constant  variance  due  to  some 
weak  responses  for  larger  cracks;  therefore,  only  hit/miss  analysis  will  be  conducted  since  a 
vs.  d  analysis  requires  constant  variance  in  the  data.  Note  that  various  transforms  were 
applied  in  an  unsuccessful  attempt  to  achieve  constant  variance. 
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FIGURE  2.  Comparison  of  empirical  and  model-generated  response  from  fastener  after  transformation  has 
been  performed.  X:  in-phase  component.  Y:  quadrature  component. 
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FIGURE  3.  Simulated  data  for  both  through  ‘crack’  and  comer  ‘crack’  responses  with  experimental  data. 


FIGURE  4.  (a)  Experimental  data  including  unflawed  responses  and  (b)  probability  density  function  for 
unflawed  responses  showing  that  the  noise  follows  a  basic  Gaussian  distribution. 


Next,  a  Monte-Carlo  study  was  conducted  to  provide  data  for  the  model-assisted  POD 
evaluation  using  the  noise  distribution  for  the  unflawed  fastener  sites,  the  corner  crack 
model,  and  the  through  crack  model.  The  Monte-Carlo  study  was  performed  using  5000 
data  points  based  on  these  assumptions.  The  results  are  displayed  in  Figure  5.  Qualitatively 
there  is  excellent  agreement  between  the  empirical  and  simulated  data.  Again,  the  only 
outlier  points  of  concern  are  the  weaker  than  expected  responses  for  some  of  the  larger 
cracks.  This  non-constant  variance  observed  with  the  larger  crack  data  is  most  likely  related 
to  the  effect  of  crack  contact  conditions,  transition  between  comer  cracks  and  through 
notches,  and  variability  of  the  crack  aspect  ratios  and  morphology. 
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FIGURE  5.  Comparison  of  simulation,  experimental  data,  and  Monte  Carlo  simulation  results.. 


For  the  quantitative  evaluation,  a  POD  curve  was  generated.  The  POD  calculations 
were  made  using  the  conventional  maximum  likelihood  approach  used  in  the  current  version 
of  MIL-HNDB-1823  [14].  Recently,  it  has  been  determined  that  the  Wald  method  for 
calculating  confidence  bounds  for  hit/miss  data  can  be  anti-conservative  [15].  The 
Loglikelihood  Ratio  Method  was  used  for  the  confidence  bound  calculations.  The  POD 
results  for  both  the  empirical  study  and  the  model-assisted  POD  approach  are  given  in 
Figure  6.  The  curve  generated  with  the  MAPOD  approach  is  within  the  95%  confidence 
bounds  of  the  empirical  curve,  demonstrating  good  agreement  between  the  two  approaches. 
Considering  that  only  the  hole  response  was  used  to  calibrate  the  model,  these  results  are 
very  encouraging.  Note,  the  confidence  bounds  on  the  model  generated  data  are  misleading 
because  the  uncertainty  in  the  input  parameters  is  not  accounted  for.  The  narrow  confidence 
bounds  actually  show  the  confidence  of  the  computations  (i.e.  the  bounds  are  inversely 
proportional  to  the  square  root  of  the  number  of  simulation  runs).  More  research  is  needed 
to  include  uncertainty  properly  into  the  analysis. 

There  are  many  opportunities  for  improvement  and  further  research.  It  would  be 
beneficial  to  know  at  what  size  the  transition  from  a  comer  crack  to  a  through  crack  occurs. 
This  transition  depends  on  the  aspect  ratio  of  the  crack;  thus,  a  study  of  aspect  ratio 
distributions  from  a  tear-down  study  would  be  beneficial.  The  effect  of  contact  conditions 
of  the  crack  faces  and  crack  morphology  on  eddy  current  response  is  still  not  resolved  and  a 
review  of  the  relevant  literature  will  be  investigated.  Future  work  is  also  needed  to  develop 
numerical  and  statistical  models  that  can  better  represent  this  observed  variability  with  large 
cracks. 


6 


target  size  (inches) 

FIGURE  6.  POD  results  for  empirical  and  MAPOD  evaluations. 


CONCLUSIONS  AND  RECOMMENDATIONS 

The  feasibility  of  model-assisted  POD  has  been  demonstrated  by  the  good  match 
between  the  empirical  and  model-assisted  POD  evaluation  results.  This  is  the  first  instance 
of  applying  a  model-assisted  POD  methodology  to  the  evaluation  of  NDE  techniques  for 
complex  structure  applications.  The  model-assisted  POD  study  only  used  data  from 
unflawed  fastener  sites  to  evaluate  the  noise  distribution  and  calibration  of  the  probe. 
Further  studies  on  the  variation  in  crack  aspect  ratio,  transition  region  between  corner  and 
through  cracks,  and  effect  of  crack  face  contact  conditions  will  enhance  model-assisted  POD 
results.  Cracks  in  between  fastener  sites  also  need  to  be  considered  through  additional 
simulated  studies  and  signal  to  noise  calculations.  New  feature  extraction  techniques 
address  limitations  based  on  location  and  number  of  fastener  sites  and  the  effect  of  steel 
fasteners  [16].  The  most  effective  way  to  calculate  confidence  bounds  for  mixed  model  and 
experimental  data  sets  is  still  undetermined  and  more  research  is  necessary.  Uncertainty  in 
input  parameters  needs  to  be  accounted  for  in  the  transfer  function  and  full-model  assisted 
approaches. 
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